Papers with compositional inference
Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)
Copied to clipboard
| Challenge: | Despite growing interest in linguistic universals, most NLI/STS studies focus on English. |
| Approach: | They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. |
| Outcome: | The proposed datasets show that pre-trained language models are insensitive to word order and case particles. |
SETI: Systematicity Evaluation of Textual Inference (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models (PLMs) have shown remarkable performance on this task, but little is known about their ability to address compositional generalization. |
| Approach: | They propose a benchmark to evaluate pre-trained language models' systematicity in the domain of textual inference. |
| Outcome: | The proposed benchmark evaluates pre-trained language models on six widely used PLMs. |
Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation (2025.findings-acl)
Copied to clipboard
Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen, David Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu
| Challenge: | Recent studies have evaluated and shown limitations in specific capabilities such as visual understanding, but a systematic evaluation of VLMs’ fundamental WM abilities remains absent. |
| Approach: | They propose a framework that assesses perception and prediction to provide an atomic evaluation of VLMs as WMs. |
| Outcome: | The proposed framework assesses perception and prediction abilities on 15 latest VLMs and compares them to human-level models. |
POLARIS: A Gödel Agent Framework for Small Language Models through Experience-Abstracted Policy Repair (2026.findings-acl)
Copied to clipboard
| Challenge: | Gödel agent Polaris makes policy level changes with small, auditable patches that persist in the policy and are reused on unseen instances within each benchmark. |
| Approach: | They propose a Gödel agent that performs policy repair via experience abstraction . Polaris makes policy level changes with small, auditable patches that persist in the policy . |
| Outcome: | The proposed agent improves on MGSM, DROP, GPQA, and LitBench models over the base policy and competitive baselines. |